Applications deployed over the web continue to increase and grow in complexity. Web technologies continue to improve allowing users to use the Internet for more than viewing static web pages. Most web pages now are interactive, with the web pages behaving and providing similar functionality to desktop applications. Web pages allow users to edit and create documents, spreadsheets, and presentations; to share information with other users including documents, videos, and images; to view or listen to streaming media; to send and receive email; to place Voice over Internet Protocol (VoIP) calls; and to play online games. The deployment of rich internet application such as ADOBE AIR, JAVAFX, MICROSOFT SILVERLIGHT, and CURL have helped developers create applications which further blur the line between the Internet and the desktop, by allowing users to run web applications as desktop applications. Such technologies allow for the advantages of performance improvements and access to local drives to be combined with online access. Further, plugins for web browsers have also helped condense the user experience to a web browser, by providing a variety of controls through the user interface of the web browser. For example, plugins available for the MOZILLA FIREFOX web browser allow a user to perform activities ranging from playing music from the user's library through the web browser interface, to managing the download of files and peer-to-peer file sharing.
In addition, desktop applications continue to increasingly make use of online access to enhance the user experience. Online access is used by video games to allow a user to play with other users in a local area network or on the Internet. Online access is further used by other applications to deliver regular updates that can be self-executing or installed manually and to allow users to search for additional content not provided by default on a desktop application.
The Internet Protocol Suite (TCP/IP) is a set of communications protocols used for the Internet and other networks, including local area networks (LAN) and wide area networks (WAN). The Internet Protocol Suite model consists of a set of layers, with each layer level providing a different level of functionality and encapsulation. Layers are usually connected to the immediate neighboring layers. For example, in a three layer model, the top layer can send a request to the middle layer, with the middle layer in turn making a request to the bottom layer in order for the middle layer to fulfill the original request by the top layer. Information is passed up and down layers, and any layer is aware of its corresponding top and bottom layers. The TCP/IP model consists of four layers, the application layer, the transport layer, the internet layer, and the link layer. The application layer is the top level layer and the link layer is the lowest level layer. Different protocols are available in each layer. Examples of protocols in the application layer include DNS, TFTP, SSL, FTP, Gopher, HTTP, MGCP, POP3, SIP, SMTP, SSH, Telnet, RTP, RTSP, and XMPP. Example protocols in the transport layer include TCP, UDP, DCCP, SCTP, RUDP, and RSVP. Example protocols of the internet layer include IP, ICMP, IGMP, and ICMPv6. Finally, example protocols of the link layer include ARP, RARP, OSPF, and NDP.
A second protocol layering model is the Open System Interconnection Reference Model (OSI Model), that is analogous to the TCP/IP model but which divides network architecture into seven layers instead of four. The seven layers, from top to bottom, are the application, presentation, session, transport, network, data-link, and physical layers. Protocols in the application layer include NNTP, SIP, SSI, DNS, FTP, Gopher, HTTP, NFS, SMPP, SMTP, and Telnet. Protocols in the presentation layer include AFP, ASCII, ICA, MIME, RDP, XDR, SSL, and TLS. Protocols in the session layer include ASP, NetBIOS, PPTP, RPC TRCP, SMPP, SCP, SSHP, and SDP. Protocols in the transport layer include TCP, UDP, PPTP, and SCTP. Protocols in the network layer include IP, ICMP, IPsec, and IGMP. Protocols in the data link layer include ARP, CSLIP, and SLIP. Protocols in the physical layer include T1, E1, 802.3 Ethernet, DSL, and 802.11a/b/g/n PHY.
A packet is a unit of data routed on a packet-switched network. A packet switched network allows for communication of data by structuring the data into formatted blocks, these being the packets. A packet consists of control information and the user data. The control information provides the network with the necessary information to deliver the data to the intended destination. This information can include source and destination addresses, error detection codes, and sequencing information. It is common for the control information to be encoded in packet headers and trailers.
Different communication protocols use different ways for distinguishing the elements and for formatting the user data. For example, the first four pieces of control information in an IPv4 protocol packet header consists of four bits that contain the version, either IPv4 or IPv6, the next four bits contain the length of the header, the next 8 bits contain the Quality of Service, and the next 16 bits contain the length of the packet in bytes.
Packet contents and the protocol that a packet belongs to can be determined by examining the packet's header. Tools such as packet sniffers or a protocol analyzer capture a packet and analyze its contents according to some specifications.
Considering the complex and multifaceted use of the Internet by a variety of users, there is a need to provide an assessment for the quality of the user experience when using the Internet.
Quality of Experience or Quality of User Experience (QoE) is supposed to be a subjective measure of a customer's experiences with a vendor. In the context of telecommunications networks, QoE is a subjective measure, from the user's perspective, of the overall value of the service provided. As such, it typically includes factors that contribute to the overall user value such as suitableness, flexibility, mobility, security, cost, personalization, and choice.
Attempts to measure and report QoE for VoIP telephone calls has focused on objective metrics that may represent a user's experience. Examples of metrics measured during live VoIP calls include information on packet loss rate, packet discard rate due to jitter, packet loss/discard burst metrics, network delay, end system delay, and signal/noise/echo level. Jitter concerns the variation in the delay between received packets. Latency is the time required for a packet to travel from one point to a second point. A low latency for a packet is desired. A packet loss occurs when one or more packets fail to reach their destination. Packet loss is caused by signal degradation over a network, oversaturated network links, corrupt packets rejected in-transit, faulty networking hardware, faulty network drivers, or normal routing routines.
Attempts have been made to measure more subjective metrics, with one of the more notable metrics for measuring QoE being referred to as Mean Opinion Score (MOS). MOS has a scale of 1-5, with 3 and above considered acceptable. The MOS is generated by averaging the results of a set of standard, subjective tests where a number of listeners listen in and rate the heard audio quality of VOIP phone calls. A listener is required to give each call a rating using the rating scheme shown in Table 1. The MOS rating is the arithmetic mean of all the individual scores.
MOS RatingDescription5Excellent—Imperceptible4Good—Perceptible but not annoying3Fair—Slightly annoying2Poor—Annoying1Bad—Very annoying
Another metric for measuring QoE is known as the R-factor, and has a scale of 1-100, with 70 and above considered acceptable. R-factor is an alternative method of assessing call quality. Scaling from 0 to 100, as opposed to the limited MOS scale of 1 to 5, makes R-factor a more precise tool for measuring voice quality. R-factor is calculated by evaluating user perceptions as well as the objective factors that affect the overall quality of a VoIP call. Some users believe R-factor to be a more objective measure of the quality of a VoIP system than MOS. MOS ratings can be further broken down by tenths to create a broader scale and compared to the R-factor on a relative basis, as set forth in the following table.
DescriptionMOS ratingR-factorVery satisfied4.3-5.0 90-100Satisfied4.0-4.380-90Some users satisfied3.6-4.070-90Many users dissatisfied3.1-3.660-70Not recommended1.0-2.6Less than 50
Performance measurements have also been proposed for media streaming and Internet connections. For example, U.S. Pat. No. 6,157,618 discloses geographically distributed data-gathering client computers, where each client computer accesses a target site, and records performance values such as load time. However, the client computers poll a central server for a target site to access, and the resulting performance values are sent to the central server for analysis. U.S. Pat. No. 7,010,598 and U.S. Patent Application No. US 2007/0271590 A1 present examples of determining quality metrics for streaming media, including assessing video and sound quality. U.S. Pat. No. 7,085,230 discloses determining quality of voice communications in order to determine the acceptability of the quality of a first communication device in comparison to a second communication device. U.S. Patent Application No. US 2008/0098446 A1 presents an example of determining metric data pertaining to the delivery of streaming content to client devices, and using the metric data to switch a client device from a current stream to a more optimum stream based on the metric data. U.S. Application No. US 2005/0089043 A1 discloses QoE metrics indicative of quality in a communication environment. A negotiation between a client and a server is used to determine which QoE metrics to use during a session between the client and the server.
A challenge faced by Internet Service Providers (ISPs) is determining how bandwidth is consumed and what is occurring with their traffic, and determining whether subscribers are satisfied with their service and are getting the expected service quality. For example, evidence suggests that peer to peer (P2P) usage remains one of the biggest consumptions of bandwidth in high bandwidth networks, which can subsequently affect the perceived QoE by other users in the network.
In addition, many ISPs either currently offer, or plan to offer, tier based subscription plans where a certain level of service is guaranteed. Guaranteed services tend to be specific to a set of activities. For example, an ISP may offer a subscription package for gamers, with service guarantees for low latency and a minimum bandwidth. In order to verify subscriber QoE for such subscription services, it is essential for ISPs to test their networks using tools that emulate the realistic subscriber behavior.
In addition, enterprises want to ensure that their networks are able to handle the network traffic load of all their users. Most importantly, they want to ensure that their networks are tested in stress situations such as power outages, failovers and other disaster recovery scenarios. Similarly, network equipment manufacturers need means to test and ensure that their edge devices meet the rigorous demands of ISPs and enterprises. The key testing aspect for providing these services is emulating user behavior in a realistic way, and determining whether service needs are being met from a subscriber point of view.
One method of testing Internet Protocol (IP) networks is to simulate subscriber activities to see how the networks respond. Typically this is accomplished by flooding the network with HTTP requests, and determining how long it takes for the requests to be handled. While this provides a certain degree of testing and information regarding the performance of the network, it does not provide a realistic emulation of typical user behavior. A subscriber does not flood the network with requests, but instead requests a web page, receives the web page, takes a certain amount of time to read the web page, clicks on a link in the webpage, etc.
Edge devices are routers, routing switches, switches, integrated access devices, multiplexers and a variety of metropolitan area network (MAN) and wide area network (WAN) access points that provide entry points into an enterprise or service provider core networks. An edge device may translate between one type of network protocol and another protocol. In general, edge devices provide access to faster, more efficient backbone and core networks. An edge device may also provide enhanced services, such as VPN support, VOIP, and Quality of Service (QoS). An edge device may or may not modify the traffic it receives, depending on the function of the edge device and the communication protocol(s) of the incoming and outgoing traffic. For example, a simple switch routes the incoming traffic without making any modifications to the incoming packets, whereas an edge device, such as a session border controller, may do some data conversions on the incoming packets before sending the modified packets.
The testing of edge devices typically requires the use of test clients and test servers. This introduces the requirement for synchronizing the behavior between the test clients and the test servers. One solution is to have no synchronization, and to simply allow the test clients to run and receive the corresponding responses from the test servers, collecting the test result information at the client and the server level. The problem with this approach is that the lack of synchronization does not allow for coordinated testing and realistic emulation of a real network. For example, a client may have instructions to send an HTTP request to a server, and the server may arbitrarily respond to the client request based on a set of instructions on the server. The server may respond 30% of the time with the correct response to the client, 10% of the time the server may respond with a response timeout, and 60% of the time with corrupt responses. However, there is no way of enforcing that the server responds to a test client A with a timeout request followed by a valid HTTP response.
A second approach is to introduce synchronization. This requires the test control to be communicated between a server and the test clients, such as by opening a second communication channel. The test control information can then be synchronized between the test clients and the test servers, so that server A knows to respond with a dropped packet to client A, and to respond with a correct HTTP response to client B. This requires a single application controlling the test control information on both sides. For example, the test control entity would inform the server to respond as X to the client Y. Alternatively, there is the issue of two communication channels going out of sync. This would require enforcing synchronization at points during the testing. For example, synchronization could be checked and corrected, if necessary, after every test client has received a response on the current iteration. Or, a synchronization check could be made every time a server received a request from a client. For example, the server could receive an FTP request, and then check through a communication channel for how to respond to the received FTP request based on the originating client. Without synchronization the testing can be invalidated, not realistic, or not reflect the intended test procedure. In addition, the communication of test control information during synchronization of the test clients and the test servers consumes bandwidth and introduces a computational overhead.
Piggybacking refers to the act of riding on the back of something else. In data transmission, piggybacking has been used to send successful receipt acknowledgements in a packet. For example, piggybacking is used in TCP connections where instead of sending an immediate acknowledgement that packet X was successfully received by entity B from entity A, entity B can wait until it must send a response to entity A, and in that response piggyback the acknowledgement for packet X or any other packet yet to be acknowledged to entity B.